1. Field of the Invention
The present invention relates to an image processing apparatus and image processing method, and particularly relates to an image processing apparatus and image processing method capable of showing three-dimensional images of a sensation closer to observations made in actual space with a system which provides three-dimension information images such as with computer graphics and the like, and more specifically relates to an image processing apparatus and image processing method capable of performing control so as to generate and show three-dimensional images of a sensation closer to observations made in actual space by means of showing images in focus and images not focused, i.e., images with out-of-focus generated, according to the point-of-view position information of the user.
2. Description of the Related Art
As a technique for showing three-dimensional images, there is a method wherein two images provided with parallax of the left and right eyes are generated, and shown to the respective left and right eyes. For example, an arrangement wherein objects and scenes in images are made to be viewed in a three-dimensional manner, by means of showing two images generated taking the parallax of the left and right eyes into consideration to the left and right eyes of a user wearing an HMD (Head Mounted Display) is in practical use.
Multiple conventional examples for showing three-dimensional images taking the parallax of the left and right eyes into consideration will be described with reference to FIGS. 1A-1C. FIG. 1A is an example primarily widely applied in the fields of video conferencing and remote-controlled robot vision, and is used to show the state of remote areas to a user in a three-dimensional manner, for example. As shown in FIG. 1A, the same object is simultaneously imaged with two cameras 101 and 102 regarding which the positional relation has been fixed. For example, the optical axes are positioned parallel and the same object or scene is taken by each, so that individual images for each are taken. The positional relation of the two cameras 101 and 102 is determined taking parallax into consideration, with the camera 101 taking images for the left eye, and the camera 102 taking images for the right eye. The images taken by each camera are displayed on two image display units, i.e., a left eye image display unit L 103 and a right eye image display unit R 104, and the user observes the image taken with camera 101 with the left eye and the image taken with camera 102 with the right eye, and thus can experience a three-dimensional sensation as if the user were actually viewing the object or scene with naked eyes.
FIG. 1B is an example of application in a field called VR (Virtual Reality) or MR (Mixed Reality), and is a configuration wherein a three-dimensional space created within a calculator can be shown to a user in a virtual manner, or the computer-synthesized three-dimensional image can be superimposed over a real-space image taken by the system shown in FIG. 1B. Three-dimensional information defining a scene or object defined in a virtual three-dimensional space within the calculator is stored within the three-dimensional information storing unit 105 shown in FIG. 1B, and a left eye image and right eye image are generated in the parallax image generating unit 106, based on this three-dimensional information. The left eye image and right eye image generated in the parallax image generating unit 106 are each two different sets of image data generate based on the stored information in the three-dimensional information storing unit 105, assuming that a virtual object for example is observed from respective visual line directions. The left eye image and right eye image are displayed on the two image display units, i.e., a left eye image display unit L 107 and a right eye image display unit R 108, and can be viewed by the user as a virtual reality (VR) image as a virtual three-dimensional image on a three-dimensional space created within the calculator, or as a mixed reality (MR) image synthesized with a real-space image taken with a camera or the like. Regarding the MR techniques, in addition to that using an image taken by a camera as described above, there is a configuration called xe2x80x9csee-throughxe2x80x9d type wherein a virtual object is superimposed on a world which the user is directly viewing, using a half-mirror or the like (e.g., xe2x80x9cTechnique for Positioning Real Worlds and Virtual Worldsxe2x80x9d, Kiyohide Sato, Image Recognition and Understanding Symposium, July 1998).
FIG. 1C is a technique used in 3D movies, and is a configuration wherein image data is prepared separately as left eye images and right eye images beforehand, stored in an image storing unit L 109 and image storing unit R 110, and synthesized to be displayed on an image display unit L 111 and an image display unit R 112.
The examples shown in FIGS. 1A-1C are configurations for showing three-dimensional images by displaying images separately for left and right eyes, but there are also methods for realizing displaying of three-dimensional images without having two image display units, i.e., with only one image display unit. This includes a method wherein the user wears glasses wherein the polarized plane of incident light to the left and right eyes becomes perpendicular, the images to be shown to the left and right eyes respectively are polarized and displayed, so as to observe images with the left and right eyes from respective points of view, a method wherein the user wears glasses capable of transmitting or shielding incident light to the left and right eyes by using liquid crystal shutters or the like, and alternately displaying images to be shown to the left and right eyes respectively in time-sequence at high speed, so as to observe images with the left and right eyes from respective points of view, and so forth.
The above conventional art represents a three-dimensional sensation by individually showing to the left and right eyes of the user two sets of image data taking parallax of the left and right eyes into consideration. However, parallax is not the only information necessary for humans to obtain a three-dimensional sensation. In the event of humans viewing an actual three-dimensional space, the eyes focus on an object at a fixation point by adjusting the thickness of the crystalline lens which serves as the lens of the eye. Thus all objects except that at the point of fixation are out of focus according to the distance thereof from the object at the fixation point, and these appear to the human eye as fuzzy or out-of-focus images. This fuzzy focus has not been taken into consideration in conventional three-dimensional image showing apparatuses, with the outline of almost all object images contained within the area image at differing distances from the fixation point being displayed clearly, giving the user viewing the images an unnatural sensation.
Thus, many conventional three-dimensional image showing apparatuses show the outline of objects contained within the area other than the fixation point at differing distances from the fixation point as clearly as that of the object at the fixation point. Accordingly, a great number of images regarding which the user is not focused upon, i.e., at positions other than the fixation point, are observed by the user with clear outlines. Such a configuration not only gives the user viewing the three-dimensional images an unnatural sensation, but also is burdensome for the eyes of the user and has been a cause of increased fatigue for the user.
The present invention has been made in light of such conventional art, and accordingly, it is an object thereof to provide an image processing apparatus, image processing method, and program providing medium, wherein three-dimensional images closer to real space can be shown to the user, by means of obtaining the fixation point position of the user observing three-dimensional images, generating out-of-focus for images which are at areas other than the fixation point in the displayed image, according to distance from the objects and scenes which are the fixation point components, and forming this into the image shown to the user.
Further, it is another object of the present invention to provide an image processing apparatus, image processing method, and program providing medium, wherein the fixation point position of the user is obtained and the image which is the object of the fixation point is brought into focus, out-of-focus is generated for images at areas other than the fixation point contained in the displayed image according to distance from the objects and scenes which are the fixation point image, and this is formed this into the displayed image shown to the user, thus enabling a state wherein the image formed on the retinas of the user approximates the state of viewing an object in real three-dimensional space with naked eyes, whereby the user observing the three-dimensional images does not experience an unnatural sensation, nor is it burdensome for the eyes but rather lessens fatigue.
The present invention has been made taking the above problems into account, and according to a first aspect of the present invention, an image processing apparatus which executes three-dimensional image generation processing comprises:
fixation point obtaining means for obtaining a fixation point position on an image display means; and
out-of-focus image generating means for generating an out-of-focus image based on three-dimensional information relating to a displayed image on the image display means;
wherein the out-of-focus image generating means takes the distance from the point of view of the display component contained in the pixel position corresponding to the fixation point position at the image display means as the standard distance;
and wherein the image processing apparatus has a configuration in which out-of-focus image generation processing is performed regarding display components on the image display means at which the difference with the standard distance exceeds a predetermined threshold value.
The out-of-focus image generation processing at the out-of-focus image generating means may be executed as different out-of-focus image generation processing depending on the distance difference which is the difference with the standard distance. Further the out-of-focus image generation processing may be performed by the out-of-focus image generating means by changing the luminance of pixels in a surrounding area containing an arbitrary pixel xe2x80x9cixe2x80x9d based on the difference between the arbitrary pixel xe2x80x9cixe2x80x9d and the standard distance, i.e., based on R(is)xe2x88x92R(i), wherein
xe2x80x9cisxe2x80x9d represents the pixel position of the fixation point on the image display means obtained by the fixation point obtaining means,
R(is) represents the standard distance which is the distance from the point of view of the pixel position xe2x80x9cisxe2x80x9d obtained based on the three-dimensional information, and
R(i) represents the distance from the point of view of the arbitrary pixel xe2x80x9cixe2x80x9d on the image display means.
Further, the image processing apparatus may have a configuration wherein the out-of-focus image generating means obtains the out-of-focus amount z(i) at the pixel xe2x80x9cixe2x80x9d on the display means by the following Expression 7;
Expression 7       ζ    ⁡          (      i      )        =                    R        ⁢                  (          is          )                    -              R        ⁢                  (          i          )                            R      ⁢              (        is        )            
wherein
xe2x80x9cisxe2x80x9d represents the pixel position on the image display means obtained by the fixation point obtaining means,
R(is) represents the standard distance which is the distance from the point of view of the pixel position xe2x80x9cisxe2x80x9d obtained based on the three-dimensional information, and
R(i) represents the distance from the point of view of the arbitrary pixel xe2x80x9cixe2x80x9d on the display means;
and wherein the radius r(i) of the area W stipulated with the pixel xe2x80x9cixe2x80x9d as the center is obtained using an arbitrarily settable parameter xe2x80x9caxe2x80x9d with the following Expression 8;
Expression 8
r(i)=axcex6(i)
and wherein the luminance value B(i) following the out-of-focus image generation processing is obtained by the following Expression 9;
Expression 9       B    ⁢          (      i      )        =                    ∑                  j          ⁢                      xe2x80x83                    ∈                      xe2x80x83                    ⁢          w                    ⁢              xe2x80x83            ⁢              I        ⁢                  (          j          )                      N  
wherein
j represents the pixels contained in the area W,
N represents the number of pixels contained in the area W, and
I(i) represents the original luminance value at the pixel xe2x80x9cixe2x80x9d;
thus executing processing for determining the luminance value B(i) as the luminance value of the pixel xe2x80x9cixe2x80x9d.
The image processing apparatus may comprise:
first image display means for displaying a left-eye image according to parallax, and second image display means for displaying a right-eye image according to parallax; and
fixation point obtaining means for obtaining a left-eye fixation point position on the first image display means and a right-eye fixation point position on the second image display means;
wherein the out-of-focus image generating means has a configuration for generating an out-of-focus image based on the three-dimensional information relating to the display image at the first image display means and the second image display means, and taking the distance from the point of view of the display component at the position corresponding to the fixation point position for the left eye at the first image display means as the left eye image standard distance and subjects the displayed image on the first image display means to out-of-focus image generation processing according to the difference between the left eye image and the standard distance, and the distance from the point of view of the display component at the position corresponding to the fixation point position for the right eye at the second image display means as the right eye image standard distance and subjects the displayed image on the second image display means to out-of-focus image generation processing according to the difference between the right eye image and the standard distance.
Here, first imaging means for taking visual line images from the visual line of the left eye and second imaging means for taking visual line images from the visual line of the right eye may comprise multiple imaging means for taking images according to parallax. Also, three-dimensional information generating means may be provided to generate three-dimensional information relating to the displayed images on the first image display means and the second image display means, based on multiple images taken by the first imaging means and the second imaging means. Further, three-dimensional information storing means may be provided for storing three-dimensional information relating to the displayed images on the first image display means and the second image display means;
wherein the out-of-focus image generating means has a configuration for executing out-of-focus image generating processing based on the three-dimensional information stored in the three-dimensional information storing means and fixation point information obtained by the fixation point obtaining means.
Further, parallax image generating means may be provided for generating two different parallax images to provide to the first image display means and the second image display means, based on the stored information in the three-dimensional information storing means;
wherein the out-of-focus image generating means has a configuration for executing out-of-focus image generating processing regarding parallax images generated by the parallax image generating means, based on the three-dimensional information stored in the three-dimensional information storing means and fixation point information obtained by the fixation point obtaining means.
Further, image storing means may be provided for storing images corresponding to each of the left and right point of view positions.
Moreover, the image display means may comprise a single image display means;
the fixation point obtaining unit may have a configuration for obtaining the fixation point average of both left and right eyes, or the fixation point of either the left or right eye, as the fixation point information;
and the out-of focus image generating means may be arranged so as to determine the distance from the point of view of the display component at the position indicated by the fixation point information received from the fixation point obtaining unit as the standard distance, and also subject the displayed image at the image display means to out-of focus image generating processing according to the difference from the standard distance, based on three-dimensional information regarding the displayed image at the image display means.
Three-dimensional information obtaining means may be provided for obtaining three-dimensional information, wherein the out-of-focus image generating means has a configuration for generating out-of-focus images using the three-dimensional information obtained from the three-dimensional information obtaining means.
Also, three-dimensional information storing means may be provided for storing three-dimensional information relating to displayed images on the single image display means, wherein the out-of-focus image generating means has a configuration for executing out-of-focus image generating processing based on the three-dimensional information stored in the three-dimensional information storing means and fixation point information obtained by the fixation point obtaining means.
Further, image generating means may be provided for generating images based on the visual line direction relating to the single image display means, based on storing information of the three-dimensional information storing means; wherein the out-of-focus image generating means has a configuration for executing out-of-focus image generating processing regarding images generated by the image generating means, based on the three-dimensional information stored in the three-dimensional information storing means and fixation point information obtained by the fixation point obtaining means. Also, image storing means may be provided for storing source images to be provided to the single image display means.
Moreover, the three-dimensional information may be made up of three-dimensional information obtained by one of the following methods:
three-dimensional information obtained by applying the stereo method based on multiple image data taken by a standard camera and a reference camera for taking the same object from different visual line directions;
three-dimensional information obtained by the multi base line stereo method using multiple reference cameras;
three-dimensional information obtained by the light-section method with slit light;
three-dimensional information obtained by a method using moire; and
three-dimensional information obtained by a distance measuring method according to pattern light irradiation.
Also, according to a second aspect of the present invention, an image processing method which executes three-dimensional image generation processing comprises:
a fixation point obtaining processing step for obtaining a fixation point position on an image display means; and
an out-of-focus image generating step for generating an out-of-focus image based on three-dimensional information relating to displayed images on the image display means;
the out-of-focus image generating step comprising:
a step for taking the distance from the point of view of the display component at a position corresponding to the fixation point position at the image display means as the standard distance; and
a step for performing the out-of-focus image generation processing regarding display components on the image display means at which the difference with the standard distance exceeds a predetermined threshold value.
The out-of-focus image generation processing in the out-of-focus image generating step may be executed as different out-of-focus image generation processing depending on the distance difference which is the difference with the standard distance.
Also, the out-of-focus image generating step may be performed by changing the luminance value of pixels in a surrounding area containing an arbitrary pixel xe2x80x9cixe2x80x9d based on the difference between the arbitrary pixel xe2x80x9cixe2x80x9d and the standard distance, i.e., based on R(is)xe2x88x92R(i), wherein
xe2x80x9cisxe2x80x9d represents the pixel position of the fixation point on the image display means obtained in the fixation point obtaining step,
R(is) represents the standard distance which is the distance from the point of view of the pixel position xe2x80x9cisxe2x80x9d obtained based on the three-dimensional information, and
R(i) represents the distance from the point of view of the arbitrary pixel xe2x80x9cixe2x80x9d on the image display means.
Further, the out-of-focus image generating step may obtain the out-of-focus amount z(i) at the pixel xe2x80x9cixe2x80x9d on the display means by the following Expression 10;
Expression 10       ζ    ⁡          (      i      )        =                    R        ⁢                  (          is          )                    -              R        ⁢                  (          i          )                            R      ⁢              (        is        )            
wherein
xe2x80x9cisxe2x80x9d represents the pixel position on the image display means obtained in the fixation point obtaining step,
R(is) represents the standard distance which is the distance from the point of view of the pixel position xe2x80x9cisxe2x80x9d obtained based on the three-dimensional information, and
R(i) represents the distance from the point of view of the arbitrary pixel xe2x80x9cixe2x80x9d;
and wherein the radius r(i) of the area W stipulated with the pixel xe2x80x9cixe2x80x9d as the center is obtained using an arbitrarily settable parameter xe2x80x9caxe2x80x9d with the following Expression 11;
Expression 11
r(i)=axcex6(i)
and wherein the luminance value B(i) following the out-of-focus image generation processing is obtained by the following Expression 12;
Expression 12       B    ⁢          (      i      )        =                    ∑                  j          ⁢                      xe2x80x83                    ∈                      xe2x80x83                    ⁢          w                    ⁢              xe2x80x83            ⁢              I        ⁢                  (          j          )                      N  
wherein
j represents the pixels contained in the area W,
N represents the number of pixels contained in the area W, and
I(i) represents the original luminance value at the pixel xe2x80x9cixe2x80x9d;
thus executing processing for determining the luminance value B(i) as the luminance value of the pixel xe2x80x9cixe2x80x9d.
A fixation point obtaining step may be provided for obtaining a left-eye fixation point position on a first image display means for displaying a left-eye image according to parallax, and for obtaining a right-eye fixation point position on a second image display means for displaying a right-eye image according to parallax;
the out-of-focus image generating step comprising:
a step for taking the distance from the point of view of the display component at the position corresponding to the fixation point position for the left eye at the first image display means as the standard distance for the left eye image;
a step for subjecting the displayed image on the first image display means to out-of-focus image generation processing according to the difference between the left eye image and the standard distance;
a step for taking the distance from the point of view of the display component at the position corresponding to the fixation point position for the right eye at the second image display means as the standard distance for the right eye image; and
a step for subjecting the displayed image on the second image display means to out-of-focus image generation processing according to the difference between the right eye image and the standard distance.
Further provided may be a step for taking visual line images from the visual line of the left eye with a first imaging means, and for taking visual line images from the visual line of the right eye with a second imaging means; and
a three-dimensional information generating step for generating three-dimensional information relating to the displayed images on the first image display means and the second image display means, based on multiple images taken by the first imaging means and the second imaging means.
The out-of-focus image generating step may execute out-of-focus image generating processing, based on the three-dimensional information stored in the three-dimensional information storing means storing three-dimensional information relating to the displayed image, at the first image display means and the second image display means, and fixation point information obtained in the fixation point obtaining step.
Also, a parallax image generating step may be provided for generating two different parallax images to provide to the first image display means and the second image display means, based on the stored information in the three-dimensional information storing means; wherein the out-of-focus image generating step executes out-of-focus image generating processing regarding parallax images generated in the parallax image generating step, based on the three-dimensional information stored in the three-dimensional information storing means and fixation point information obtained in the fixation point obtaining step.
Further, the image display means may comprise a single image display means;
wherein the fixation point obtaining step obtains the fixation point average value of both left and right eyes, or the fixation point of either the left or right eye, as the fixation point information;
and wherein the out-of focus image generating step comprises:
a step for determining the distance from the point of view of the display component at the position indicated by the fixation point information obtained in the fixation point obtaining step as the standard distance, based on three-dimensional information relating to the displayed image on the image display means;
and a step for subjecting the displayed image at the image display means to out-of focus image generating processing according to the difference from the standard distance.
Further provided may be a three-dimensional information obtaining step for obtaining three-dimensional information, wherein the out-of-focus image generating step generates out-of-focus images using the three-dimensional information obtained in the three-dimensional information obtaining step, and also provided may be an image generating step for generating images based on the visual line direction relating to the single image display means, based on storing information of the three-dimensional information storing means; wherein the out-of-focus image generating step executes out-of-focus image generating processing regarding images generated in the image generating step, based on the three-dimensional information stored in the three-dimensional information storing means and fixation point information obtained in the fixation point obtaining step.
Further, according to a third aspect of the present invention, a program providing medium which tangibly provides computer programs for performing processing on a computer system to execute three-dimensional image processing, comprises:
a fixation point obtaining step for obtaining a fixation point position on an image display means; and
an out-of-focus image generating step for generating an out-of-focus image based on three-dimensional information relating to displayed images on the image display means;
the out-of-focus image generating step comprising:
a step for taking the distance from the point of view of the display component at the position corresponding to the fixation point position at the image display means as the standard distance; and
a step for performing the out-of-focus image generation processing regarding display components on the image display means at which the difference with the standard distance exceeds a predetermined threshold value.
The program providing medium according to the third aspect of the present invention is a medium for providing computer programs in a computer-readable format to general-purpose computer systems capable of executing various types of program code, for example. The form of the medium is not particularly restricted, and may be storage mediums such as CDs, FDs, MOs, etc., or may be transfer mediums such as a network, and so forth.
Such a program providing medium defines the structural or functional cooperation relation between the computer program and the providing medium, for realizing the predetermined functions of the computer program on the computer system. In other words, installing the computer program into the computer system via the providing medium causes the cooperating action to be exhibited upon the computer system, thereby obtaining the same actions and advantages as with the first aspect of the present invention.
Further objects, characteristics and advantages of the present invention will become apparent from the detailed description of the later-described embodiments of the present invention and of the attached drawings.